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Prospective Registration and Outcome-Reporting 
Bias in Randomized Controlled Trials of Eczema 
Treatments: A Systematic Review 

Helen Nankervis 1 , Akerke Baibergenova 2 , Hywel C. Williams 1 and Kim S. Thomas 1 

We assessed completeness of trial registration and the extent of outcome-reporting bias in published 
randomized controlled trials (RCTs) of eczema (atopic dermatitis) treatments by surveying all relevant RCTs 
published from January 2007 to July 2011 located in a database called the Global Resource of Eczema Trials 
(GREAT). The GREAT database is compiled by searching six bibliographic databases, including EMBASE and 
MEDLINE. Out of 109 identified RCTs, only 37 (34%) had been registered on an approved trial register. Only 18 
out of 109 trials (17%) had been registered "properly" in terms of submitting the registration before the trial end 
date and nominating a primary outcome. The proportion of "any registered" and "properly registered" RCTs 
increased from 19% and 10% in 2007 to 57% and 36% in 2011, respectively. Assessment of selective outcome- 
reporting bias was difficult even among the properly registered trials owing to unclear primary outcome 
description especially with regard to timing. Only 5 out of the 109 trials (5%) provided enough information for 
us to be confident that the outcomes reported in the published trial were consistent with the original 
registration. Adequate trial registration and description of primary outcomes for eczema RCTs is currently poor. 

Journal of Investigative Dermatology (2012) 132, 2727-2734; doi:1 0.1 038/ji<±201 2.231; published online 26 July 2012 



INTRODUCTION 

Outcome-reporting bias in a clinical study is defined as 
selective reporting of a subset of the study findings based on 
the significance and direction of the results. Selective 
reporting of outcomes has been well recognized in general 
medicine trials where around 40-62% of studies have been 
found to have at least one primary outcome that was 
changed, introduced, or omitted (Dwan et al., 2008). The 
effects of such distortion of what was planned in a study are 
potentially serious for clinical practice. In their review of the 
effects of outcome-reporting bias in systematic reviews, 
Kirkham et al. (2010) found that of 42 meta-analyses with a 
statistically significant result, 8 (19%) became nonsignificant 
after adjusting for outcome-reporting bias and 11 (26%) 
would have overestimated the treatment effect by ^20%. 
The problem of outcome-reporting bias has not been studied 
among randomized controlled trials (RCTs) in the field of 
dermatology. 
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One of the main ways of reducing selective outcome- 
reporting bias is to require all trial investigators to register 
details of their trial in a publicly accessible register before the 
trial recruitment starts. Those reading the final trial report can 
then go back to the original trial registration to check whether 
what was highlighted as the primary outcome measure in the 
published report corresponded to that in the original 
registration. In 2004, the International Committee of Medical 
Journal Editors (ICMJE) initiated a policy requiring investiga- 
tors to register their trials into a clinical trial registry before 
participant enrollment begins as a condition of publication in 
one of their journals (DeAngelis et al., 2004). This policy 
came into effect in July 2005. A number of dermatology 
journals followed this policy, including the British Journal of 
Dermatology and the Journal of Investigative Dermatology in 
2005 (Williams and Stern, 2005; Ormerod and Williams, 
2005). A number of trial registers sprang up in the early 
1990s, containing different core items, culminating in the 
World Health Organization (WHO) hosting an International 
Clinical Trial Registry Platform of approved trial registries 
(http://www.who.int/ictrp/en/, accessed 23 November 2011) 
that fulfilled key quality criteria and collected a minimum set 
of key study domains. Around the same time, dermatology 
journals also encouraged authors to report all of the 
important study features included in the Consolidated 
Standards of Reporting Trials Statement (http://www.consort- 
statement.org/, accessed 23 November 2011). 

This study sought to assess the extent to which out- 
come-reporting bias is evident in the field of dermatology, 
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using published RCTs of eczema treatments in people with 
eczema (atopic dermatitis) as an example. The study had 
three objectives: 

(1) To assess the proportion of eczema treatment RCTs 
registered. 

(2) To see whether the lack of registration was associated 
with differences in risk of bias, sample size, and funding 
source. 

(3) To assess the level of possible outcome-reporting bias of 
the primary outcome in eczema treatment trials. 

Our hypotheses were that (1) less than half of eczema 
RCTs would have been registered; (2) quality of reporting of 
trials without preregistration would be poorer; and (3) even in 
the trials that were properly registered, selective outcome- 
reporting bias may still exist, i.e., there may be discrepancies 
between the primary outcome that was registered and the 
primary outcome reported in the published trial report. 

RESULTS 

A total of 122 eczema RCTs were published between 1 
January 2007 and 31 July 2011 and included in the Global 
Resource of Eczema Trials (GREAT) database (Figure 1). 
Twelve studies were excluded because they had published 
abstracts, but it was not possible to identify a full trial 
publication. Two publications came from the same study, 
with the latter publication being a follow-up of the original 
trial (van der Aa et al., 2010). Only the first publication, 
describing the original trial, was included in this study. 



Search result (n= 122) 

RCTs in GREAT database and 
published between 1 January 2007 
and 31 July 2011 









RCT with full publication (n = 109) 






f i 
Registered RCTs (n = 37) 








Properly registered RCTs (n = 18) 



Excluded (n= 12) 

Abstracts only (n = 1 1) 
Follow-up of the same original 
trial (n= 1) 



Not properly registered RCTs 
("=19) 

No primary outcome in the 
registration record (n = 1) 
RCT was registered after study 
completion (n = 18) 



Figure 1. Flowchart of trial selection. GREAT, Global Resource of Eczema 
Trials; RCT, randomized controlled trial. 



Levels of registration of eczema treatment trials 

Of the remaining 109 included studies, only 37 (34%) had 
been registered in any form as shown in the study flowchart 
in Figure 1. In 9 of these 37 trials, the trial registration was 
done more than a year after the study had been completed 
and published (the longest time interval was almost 3 years 
after study completion), and thus could not be defined as 
"properly registered". Only 20 (54%) of the 37 registered 
studies indicated their trial registration number in the final 
trial publication. 

Among the 37 studies with a registered protocol, 18 
studies (49%) had "properly registered" according to our 
definition. Although there has been an increase in trial 
registration from 2007 onward, the proportion of "properly 
registered" trials was still low, reaching a maximum of 
29% (4/14) in the first half of 2011. The increasing trend in 
the proportion of registered eczema treatment trials was 
significant (P= 0.003) (Figure 2). 

Differences between trials with and without registration 

Although registered trials had a larger median sample size of 
70 people (interquartile range 37.5-192) compared with 60 
people (interquartile range 30-104.5) for trials without 
registration, this difference was not statistically significant 
(P= 0.405). Nonregistered trials were significantly less likely 
to specify their funding source in the subsequent publication 
than registered trials (3% vs 30%, P= 0.001), and a post-hoc 
analysis looking at choice of comparator showed that 
nonregistered trials were no more likely to include a placebo 
comparator than registered trials (70% vs 57%, P=0.188). 

Overall, the differences in the estimated risk of bias were 
not statistically significant, with the exception of allocation 
concealment domain, where registered trials scored margin- 
ally better: 24% (9/37) were at low risk of bias compared with 
8% (6/72) for nonregistered trials, P=0.04 (Figure 3). 

Outcome reporting in properly registered trials and their 
registration records 

The most common discrepancy was missing time-frame 
information for measurement of the primary outcome: in 




2007 2008 2009 2010 2011 



■ Proportion of trials registered ■ Proportion of trials properly registered 

Figure 2. Proportion of eczema trials that were "registered" and "properly 
registered" among those published from January 2007 to July 2011. Total 
convenience sample = 109 trials. 
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Figure 3. Risk of bias among trials with and without registration (% at low, 
unclear, and high risk of bias). 



8 out of 18 trials, registration records did not state the 
timing of the primary outcome measurement. In addition, the 
description of the primary outcome was often vague and 
unclear: this occurred in 5 out of the 1 8 "properly registered" 
trials (Table 1). For example, a primary outcome might be 
described as "efficacy in treating exacerbations" or "stable 
remission", without further detail as to how this was to 
be measured or assessed. 

Assessment of outcome-reporting bias among properly 
registered trials 

The lack of detail and clarity of reporting for the outcomes 
of registered trials both from the registration record and from 
the published trial report makes the objective of assessing 
outcome-reporting bias in eczema treatment trials difficult. 
One study introduced a new statistically significant primary 
outcome that was not present in the registration record 
(original primary outcome was not significant) (Hon et al., 
2007), one study changed the eczema severity score from 
SCORAD (Scoring of Atopic Dermatitis) to SASSAD (Six Area 
Six Sign Atopic Dermatitis Score) (Gambichler et al., 2009), 
and one study changed the time frame for assessment of the 
primary outcome from 26 weeks to 4 weeks (Frankel et al., 
2011). None of these latter changes resulted in a significant 
result being reported. 

DISCUSSION 
Main findings 

Our study has shown that the proportion of published eczema 
RCTs registered on an approved trial registry increased from 
1 9% in 2007 to 57% in 201 1 . The fact that over half of the 



RCTs published in 2011 were registered is cause for some 
optimism. However, the proportion of trials that were 
"properly registered", lodging details of the proposed trial 
design before the end of the study, or within 1 2 months of the 
recorded trial start date and indicating the primary outcome 
in the registration, was considerably lower. Taking into 
account that our definition of "properly registered" was a 
very generous one, this is a particularly sobering result. 
Overall, of the 109 eczema RCTs examined, only 5 trials 
were registered a priori, gave sufficient detail about the 
primary outcome to judge adherence to the original registra- 
tion record, and reported the primary outcome in line with 
the original registration (or included an explanation as to why 
this had changed). That only 5% of the recent eczema trials 
were registered correctly and with enough detail to assess 
outcome-reporting bias for the primary outcome, coupled 
with the observation that nonregistered or incompletely 
registered studies fail to highlight the shortcomings of such 
omissions, is bad science and a potential waste of resources 
(Chalmers, 1990; Chalmers and Glasziou, 2009). 

Even when investigators had preregistered their trial, 
46% failed to include details of their trial registration in the 
published trial report, suggesting that investigators and 
journal editors do not yet appreciate the importance of such 
information in their trial reports. The revised version of the 
Consolidated Standards of Reporting Trials Statement state- 
ment for guidance on the reporting of RCTs now explicitly 
recommends that details of trial registration be reported fully, 
and this should improve trial reporting in the future 
(http://www.consort-statement.org/, accessed 23 November 
2011). 

The fact that some investigators chose to preregister their 
trial could be an indicator of trial quality, and this was 
explored using the four key domains known to be associated 
with a high risk of bias (Higgins and Altman, 2008). In this 
sample of eczema RCTs, there was a suggestion that trial 
quality might be improved in registered trials, but this was 
only significant for the domain of allocation concealment, 
and is possibly limited by the modest sample size of our 
survey. 

With regard to the hypothesis that eczema trials are 
subject to outcome-reporting bias, we found some evidence 
of discrepancies between trial registration and trial reports. 
However, these were generally as a result of unclear and 
nonspecific trial registration information rather than clear 
signs of biased reporting. In general, the number of correctly 
registered trials was so low that firm conclusions are difficult. 
Greater efforts could certainly be made to provide more 
detail regarding the primary outcome and its timing within 
trial registry records. It is hoped that reviews such as this one 
will be helpful in informing trialists of the importance and 
relevance of detailed information being lodged in the trial 
registries. It is also important that changes to the registration 
record subsequent to trial commencement be fed into the trial 
registries in a timely manner, in order to maintain trans- 
parency throughout. The concept of threaded publications, 
enabling the tracking of clinical research studies from 
inception and the linking of all resulting publications, 
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including the raw data where this is available, is another 
emerging tool to ensure consistency between what was 
planned and what was done (Faure and Hrynaszkiewicz, 
2011). 

Comparison with other studies 

Overall, the rates for registration of eczema RCTs were lower 
than those reported for other surveys of general medical 
journals and larger specialities. Mathieu et al. (2009) looked 
at RCTs indexed in 2008 in 10 general medicine and 
specialty journals (cardiology, rheumatology, and gastroen- 
terology); of the 323 included trials, 45.5% were adequately 
registered, 13.9% were registered after the completion of 
study, and 12% were registered with no or unclear 
description of the primary outcome. 

A systematic review of studies examining the impact of 
selective outcome reporting in RCTs identified five studies 
and compared the trial publication with trial registration 
record. It found that between 40 and 62% of trials had at least 
one primary outcome that had been changed, introduced, or 
omitted (Dwan et al., 2008). These rates are higher than the 
rates found in the current study, which might be a reflection 
of improvements in trial reporting subsequent to the 
introduction of the ICMJE policy. Alternatively, it is possible 
that higher rates of outcome-reporting bias would have been 
found if more of the eczema trials had been registered, thus 
allowing a comparison of the published results with the trial 
registration. It is conceivable that to date only the better- 
quality eczema trials have been registered appropriately. 

Strengths and limitations of our study 

This study used the GREAT database to identify and assess all 
eczema trials published over the past four and a half years. 
Using this global collection of appraised eczema trials 
allowed for the speedy completion of this review and the 
easy identification of all relevant RCTs that had been 
identified through a sensitive search of several bibliographic 
databases. Including all trials on a particular topic, rather 
than just those reported in specific journals or as a random 
sample of published trials, meant that our study is less open to 
selection bias. Eczema is one of the most common skin 
conditions and is commonly researched. More than 250 RCTs 
of eczema treatments have been published in the past 10 
years alone (www.greatdatabase.org.uk, accessed August 
2011). It is possible that for other common skin conditions, 
the proportion of trials registered and possible outcome- 
reporting bias may be different, which makes the results of 
this study difficult to generalize to the field of dermatology 
research as a whole. 

Our definition of a properly registered trial as one that was 
registered before the end of the study was a generous one, 
given that the ICMJE requires that all clinical trials need to be 
registered in an approved publicly accessible clinical trial 
register before patient recruitment begins. It is likely that the 
bottom-line figure of only 5% of all recent eczema trials that 
have been properly registered would have been even worse if 
this more stringent definition had been applied, although in 
many cases it was impossible for us to tell when recruitment 
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began and when the study ended. On the point of timing, it is 
worth noting that some trial ists registered their trials after the 
study had been completed, which defeats the whole purpose 
of prospective trial registration as a tool to prevent cheating. 
Our sample size was relatively small, which affected the 
overall precision of our estimates of "properly registered" 
RCTs. 

Finally, this review limited its scope to an evaluation of the 
primary outcomes assessed in trials. Others have reported 
considerable outcome-reporting bias for secondary outcomes, 
particularly in relation to the preferential reporting of 
statistically significant secondary outcomes (Chan eta\. f 2004). 

Implications of our findings 

The introduction of mandatory trial registration as required by 
the Journal of Investigative Dermatology is a great opportu- 
nity to improve the quality and truthfulness of trial reporting, 
which will in turn lead to better clinical decision-making by 
reducing the prevalence of spurious and misleading results 
and reduction in research wastage (Chalmers and Glasziou, 
2009). Although the publication bias of whole studies is 
relatively well known, the phenomenon of selective out- 
come-reporting bias within studies is possibly less well 
understood within the dermatology clinical community 
(Chan and Altman, 2005). 

Ideally, deviations from the trial registration record should 
be described in the published articles so that readers can 
interpret the results in full knowledge of the changes made. 
For the full benefits of trial registration to be realized, it is 
important that all investigators, funders, journal editors, peer 
reviewers, readers, and the public have an active role in 
making full use of trial registration information and high- 
lighting the need for transparent trial reporting. Journal 
referees and readers of clinical trials especially should make 
more use of scrutinizing trial registries to note whether a 
clinical trial has truly been registered prospectively, and 
whether the outcomes reported in the paper are consistent 
with what was planned. Those conducting systematic reviews 
are also in a good position to check on and comment on trial 
registration of included trials, and to assess whether selective 
outcome-reporting bias was likely to have occurred. 

Conclusion 

Adequate trial registration for eczema RCTs is poor. 
Registration of all trials in a publicly accessible database is 
a critical step toward ensuring the transparent reporting of 
clinical trial results that affect health care. 

MATERIALS AND METHODS 

A systematic search for RCTs of eczema treatments that had been 
published since January 2007 was conducted. For all identified trials, 
the WHO International Clinical Trial Registry Platform was searched 
for proof of approved trial registration. 

Searching for published eczema trials 

We used a convenience sample of all published eczema trials 
that have been captured in the GREAT database (http://www. 
greatdatabase.org.uk, accessed August 2011). This open-access 



resource brings together information on all RCTs of eczema 
treatments published since the beginning of 2000 in order to 
facilitate future methodological research and systematic reviews. 
The GREAT database uses the Cochrane Highly Sensitive Search 
Strategy for identifying randomized trials in MEDLINE: sensitivity- 
maximizing version (2008 revision); Ovid format (Lefebvre et al., 
2009) combined with the terms eczema, atopic eczema, atopic 
dermatitis, infantile eczema, childhood eczema, neurodermatitis, 
and Besnier's prurigo. Similar searches combined with the same list 
of eczema terms are used to search the EMBASE, CINAHL, AMED, 
and LILACS databases. The Cochrane Library (www.thecochraneli- 
brary.co.uk) and the Cochrane Skin Group Specialised register are 
also searched using the same eczema terms. The search runs from 
the beginning of 2000 and is updated regularly. 

Inclusion criteria for the present study 

Allowing 1 .5 years from the ICMJE policy of trial registration coming 
into effect, we included all eczema treatment RCTs contained in the 
GREAT database that had been published between 1 January 2007 
and 31 July 201 1 . Whether or not these trials contained safety data 
was not relevant to this study and thus did not form part of the 
inclusion or exclusion criteria. We excluded RCTs that were 
published in abstract form only. If both an abstract and a full paper 
were found, only the full paper was included. Details of all trials 
included in this study are listed as an online table (Supplementary 
Table S1 online). 

Identifying trial registration 

If the trial registration ID number was stated in the final publication, 
we used this number to find the trial registration. Where this number 
was not stated, we tried to obtain the registration record using the 
WHO International Clinical Trial Registry Platform, which searches 
a number of different trial registries throughout the world (see 
Table 2). 



Table 2. The list of data providers of the WHO ICTRP 
search portal 1 

Australian New Zealand Clinical Trials Registry (ANZCTR) 
Chinese Clinical Trial Register (ChiCTR) 

Clinical Research Information Service (CRiS), Republic of Korea 
ClinicalTrials.gov 

Clinical Trials Registry— India (CTRI) 

Cuban Public Registry of Clinical Trials (RPCEC) 

German Clinical Trials Register (DRKS) 

Iranian Registry of Clinical Trials (IRCT) 

ISRCTN.org 

Japan Primary Registries Network (JPRN) 
Pan African Clinical Trial Registry (PACTR) 
Sri Lanka Clinical Trials Registry (SLCTR) 
The Netherlands National Trial Register (NTR) 

^Source: World Health Organization website http://www.who.int/ictrp/ 
search/data_providers/en/i ndex.html (last accessed 12 August 2011). 
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Two people searched for trial registration independently (AB and 
HN) using a combination of key words: "eczema" or "atopic 
dermatitis", plus keywords describing the trial's interventions, such 
as "pimecrolimus" or "probiotics". All registration records that met 
the search criteria were reviewed to find the one that matched the 
published trial using key information supplied in the record (e.g., the 
name of principal investigator, funding source, and trial design). 
As a rule, we did not spend more than 15-20 minutes looking 
for each record, on the grounds that trial registration should be easy 
and quick to identify. 

Distinguishing between "registered" and 
"properly registered" trials 

All eczema treatment trials in the sample from the GREAT database 
registered on the WHO International Clinical Trial Registry Platform 
were included in the number of "registered" trials. This provides the 
overall level of trial registration for eczema trials, regardless of 
whether or not they fulfilled the guidelines for trial registration 
correctly. In addition, we identified a "properly registered" subset of 
these trials, which could be used to assess outcome-reporting bias. 
We defined "properly registered" trials as those trials in which the 
primary outcome was explicitly stated in the registration document 
and in which the trial was registered not later than the end of the 
study (last patient visit). In cases in which the start of participant 
enrollment was stated, but an end date was not recorded, we 
considered a trial to be properly registered if the registration was 
lodged within 12 months of the study start date. Only "properly 
registered" trials were included in the evaluation of outcome- 
reporting bias. 

To take into account the amendments and possible changes that 
could have taken place after initial trial registration but before data 
lock, we used the URL provided on the WHO trial registration 
webpage to go to the source record for each of the "properly 
registered" trials in the primary register and looked for additional 
information or updates regarding the trial. 

Comparison between registered and nonregistered trials 

We compared registered and nonregistered trials with respect 
to several characteristics, including the number of participants 
randomized, funding, and quality of reporting. The quality of 
reporting was assessed using the Cochrane Collaboration's 
risk of bias tool (Higgins and Altman, 2008) in four different 
domains: randomization method, allocation concealment, blinding 
(of participants, personnel, and outcome assessors), and complete- 
ness of outcome data. Each trial was assessed for high, 
low, or unclear risk of bias for each of these domains using the 
revised terms suggested in the updated version of the Cochrane 
Handbook (Higgins et al., 2011) to improve interpretation of the 
risk of bias. 

Comparison of primary outcome between publication and 
registration among properly registered trials 

For each properly registered trial, we reviewed and compared 
primary outcomes reported in the publication with those stated in 
the registration. Both the primary outcome and the time frame for 
analysis were recorded. We chose to limit the review of outcome 
bias to the primary outcome for this first study, because of the 
importance of primary outcomes in determining the success or 



otherwise of a study and because detailed analysis of secondary 
outcomes was beyond the scope of our available resources. 

We defined a discrepancy as follows: (1) when the primary 
outcome in the published report was different from that in the 
registration and (2) when the time frame for assessing the primary 
outcome in the published report was different from the registration. 

Sample size calculation 

Sample size was determined by the number of available trials that 
had been published since the introduction of the ICMJE guidelines. 
We also estimated that we would need around 100 studies in order 
to estimate the proportion of registered studies (which we hypothe- 
sized would be around 50%) to within 1 0 percentage points. All eczema 
RCTs published in the past four and a half years were included. 

Data assessment 

The following comparisons were made: 

(1) The proportion of trials registered. 

(2) The proportion of trials properly registered versus the propor- 
tion registered in any form. 

(3) The proportion of registered trials with the registration number 
stated in the final publication. 

(4) The average sample size of registered trials versus nonregistered 
trials. 

(5) The proportion of trials that declared a funding source. 

(6) The proportion of registered versus nonregistered trials with a 
placebo comparator. 

(7) The number of trials with a low/unclear/high risk of bias for 
each of the four domains for registered trials versus non- 
registered trials. 

(8) The number of properly registered trials with a missing time 
frame for the primary outcome. 

(9) The number of properly registered trials with a vague and/or 
unclear primary outcome. 

Statistical analysis 

We used frequency and percentages for categorical variables, and 
mean with SD or median and interquartile ranges for continuous 
variables, as appropriate. Proportions were compared using x 2 and 
Fisher's exact tests where appropriate. Trends over time were assessed 
using % 2 for trend. Continuous variables were compared using 
f-test and Mann-Whitney U-test for nonparametric data. P<0.05 
(two-tailed) was considered statistically significant. All statistics 
were calculated using SPSS version 16 (IBM, www.ibm.com) except 
% 2 for trend, which was calculated using StatsDirect (Stats Direct, 
www.statsdirect.com). The protocol for this study was posted on the 
Centre of Evidence Based Dermatology website (http://www.notu' n- 
gham.ac.uk/dermatology) at the beginning of the study in September 
201 1 and before data analysis. 
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